การรวบรวม Log จาก EC2 Instance โดยใช้ CloudWatch Agent บน Amazon Linux 2023

sudo su -
yum update -y

yum install rsyslog -y
systemctl enable rsyslog --now

less /var/log/messages

Aug 14 03:21:15 ip-172-31-8-142 kernel: Linux version 6.1.102-108.177.amzn2023.x86_64 (mockbuild@ip-10-0-62-55) (gcc (GCC) 11.4.1 20230605 (Red Hat 11.4.1-2), GNU ld version 2.39-6.amzn2023.0.10) #1 SMP PREEMPT_DYNAMIC Wed Jul 31 10:18:50 UTC 2024
Aug 14 03:21:15 ip-172-31-8-142 kernel: Command line: BOOT_IMAGE=(hd0,gpt1)/boot/vmlinuz-6.1.102-108.177.amzn2023.x86_64 root=UUID=2277f5ea-ebeb-42da-a2e1-3b9cf1c1bca9 roconsole=tty0 console=ttyS0,115200n8 nvme_core.io_timeout=4294967295 rd.emergency=poweroff rd.shell=0 selinux=1 security=selinux quiet
Aug 14 03:21:15 ip-172-31-8-142 kernel: KASLR enabled
Aug 14 03:21:15 ip-172-31-8-142 kernel: BIOS-provided physical RAM map:
Aug 14 03:21:15 ip-172-31-8-142 kernel: BIOS-e820: [mem 0x0000000000000000-0x000000000009ffff] usable
Aug 14 03:21:15 ip-172-31-8-142 kernel: BIOS-e820: [mem 0x0000000000100000-0x00000000314cdfff] usable
Aug 14 03:21:15 ip-172-31-8-142 kernel: BIOS-e820: [mem 0x00000000314ce000-0x000000003174dfff] reserved
Aug 14 03:21:15 ip-172-31-8-142 kernel: BIOS-e820: [mem 0x000000003174e000-0x000000003175dfff] ACPI data
Aug 14 03:21:15 ip-172-31-8-142 kernel: BIOS-e820: [mem 0x000000003175e000-0x00000000317ddfff] ACPI NVS
Aug 14 03:21:15 ip-172-31-8-142 kernel: BIOS-e820: [mem 0x00000000317de000-0x000000003d77bfff] usable
Aug 14 03:21:15 ip-172-31-8-142 kernel: BIOS-e820: [mem 0x000000003d77c000-0x000000003d7fffff] reserved
Aug 14 03:21:15 ip-172-31-8-142 kernel: NX (Execute Disable) protection: active
Aug 14 03:21:15 ip-172-31-8-142 kernel: extended physical RAM map:
Aug 14 03:21:15 ip-172-31-8-142 kernel: reserve setup_data: [mem 0x0000000000000000-0x000000000009ffff] usable
Aug 14 03:21:15 ip-172-31-8-142 kernel: reserve setup_data: [mem 0x0000000000100000-0x000000002f6be017] usable
Aug 14 03:21:15 ip-172-31-8-142 kernel: reserve setup_data: [mem 0x000000002f6be018-0x000000002f6c6e57] usable
Aug 14 03:21:15 ip-172-31-8-142 kernel: reserve setup_data: [mem 0x000000002f6c6e58-0x00000000314cdfff] usable
Aug 14 03:21:15 ip-172-31-8-142 kernel: reserve setup_data: [mem 0x00000000314ce000-0x000000003174dfff] reserved
Aug 14 03:21:15 ip-172-31-8-142 kernel: reserve setup_data: [mem 0x000000003174e000-0x000000003175dfff] ACPI data
Aug 14 03:21:15 ip-172-31-8-142 kernel: reserve setup_data: [mem 0x000000003175e000-0x00000000317ddfff] ACPI NVS
Aug 14 03:21:15 ip-172-31-8-142 kernel: reserve setup_data: [mem 0x00000000317de000-0x000000003d77bfff] usable
Aug 14 03:21:15 ip-172-31-8-142 kernel: reserve setup_data: [mem 0x000000003d77c000-0x000000003d7fffff] reserved
Aug 14 03:21:15 ip-172-31-8-142 kernel: efi: EFI v2.70 by EDK II
Aug 14 03:21:15 ip-172-31-8-142 kernel: efi: SMBIOS=0x3166a000 ACPI=0x3175d000 ACPI 2.0=0x3175d014 MEMATTR=0x2fe10a98
Aug 14 03:21:15 ip-172-31-8-142 kernel: SMBIOS 2.7 present.
Aug 14 03:21:15 ip-172-31-8-142 kernel: DMI: Amazon EC2 t3a.micro/, BIOS 1.0 10/16/2017
Aug 14 03:21:15 ip-172-31-8-142 kernel: Hypervisor detected: KVM
Aug 14 03:21:15 ip-172-31-8-142 kernel: kvm-clock: Using msrs 4b564d01 and 4b564d00
Aug 14 03:21:15 ip-172-31-8-142 kernel: kvm-clock: using sched offset of 4257464174 cycles
Aug 14 03:21:15 ip-172-31-8-142 kernel: clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns
Aug 14 03:21:15 ip-172-31-8-142 kernel: tsc: Detected 2199.990 MHz processor
Aug 14 03:21:15 ip-172-31-8-142 kernel: last_pfn = 0x3d77c max_arch_pfn = 0x400000000
/var/log/messages

yum install amazon-cloudwatch-agent -y
yum install collectd -y

sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-config-wizard

[root@ip-xx-xx-xx-xx ~]# sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-config-wizard
================================================================
= Welcome to the Amazon CloudWatch Agent Configuration Manager =
=                                                              =
= CloudWatch Agent allows you to collect metrics and logs from =
= your host and send them to CloudWatch. Additional CloudWatch =
= charges may apply.                                           =
================================================================
On which OS are you planning to use the agent?
1. linux
2. windows
3. darwin
default choice: [1]:

Trying to fetch the default region based on ec2 metadata...
I! imds retry client will retry 1 timesAre you using EC2 or On-Premises hosts?
1. EC2
2. On-Premises
default choice: [1]:

Which user are you planning to run the agent?
1. cwagent
2. root
3. others
default choice: [1]:
2
Do you want to turn on StatsD daemon?
1. yes
2. no
default choice: [1]:

Which port do you want StatsD daemon to listen to?
default choice: [8125]

What is the collect interval for StatsD daemon?
1. 10s
2. 30s
3. 60s
default choice: [1]:

What is the aggregation interval for metrics collected by StatsD daemon?
1. Do not aggregate
2. 10s
3. 30s
4. 60s
default choice: [4]:

Do you want to monitor metrics from CollectD? WARNING: CollectD must be installed or the Agent will fail to start
1. yes
2. no
default choice: [1]:

Do you want to monitor any host metrics? e.g. CPU, memory, etc.
1. yes
2. no
default choice: [1]:

Do you want to monitor cpu metrics per core?
1. yes
2. no
default choice: [1]:

Do you want to add ec2 dimensions (ImageId, InstanceId, InstanceType, AutoScalingGroupName) into all of your metrics if the info is available?
1. yes
2. no
default choice: [1]:

Do you want to aggregate ec2 dimensions (InstanceId)?
1. yes
2. no
default choice: [1]:

Would you like to collect your metrics at high resolution (sub-minute resolution)? This enables sub-minute resolution for all metrics, but you can customize for specific metrics in the output json file.
1. 1s
2. 10s
3. 30s
4. 60s
default choice: [4]:

Which default metrics config do you want?
1. Basic
2. Standard
3. Advanced
4. None
default choice: [1]:

Current config as follows:
{
        "agent": {
                "metrics_collection_interval": 60,
                "run_as_user": "root"
        },
        "metrics": {
                "aggregation_dimensions": [
                        [
                                "InstanceId"
                        ]
                ],
                "append_dimensions": {
                        "AutoScalingGroupName": "${aws:AutoScalingGroupName}",
                        "ImageId": "${aws:ImageId}",
                        "InstanceId": "${aws:InstanceId}",
                        "InstanceType": "${aws:InstanceType}"
                },
                "metrics_collected": {
                        "collectd": {
                                "metrics_aggregation_interval": 60
                        },
                        "disk": {
                                "measurement": [
                                        "used_percent"
                                ],
                                "metrics_collection_interval": 60,
                                "resources": [
                                        "*"
                                ]
                        },
                        "mem": {
                                "measurement": [
                                        "mem_used_percent"
                                ],
                                "metrics_collection_interval": 60
                        },
                        "statsd": {
                                "metrics_aggregation_interval": 60,
                                "metrics_collection_interval": 10,
                                "service_address": ":8125"
                        }
                }
        }
}
Are you satisfied with the above config? Note: it can be manually customized after the wizard completes to add additional items.
1. yes
2. no
default choice: [1]:

Do you have any existing CloudWatch Log Agent (http://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/AgentReference.html) configuration file to import for migration?
1. yes
2. no
default choice: [2]:

Do you want to monitor any log files?
1. yes
2. no
default choice: [1]:

Log file path:
/var/log/messages
Log group name:
default choice: [messages]
tinnakorn-cwagent-ec2/var_log_messages
Log group class:
1. STANDARD
2. INFREQUENT_ACCESS
default choice: [1]:

Log stream name:
default choice: [{instance_id}]

Log Group Retention in days
1. -1
2. 1
3. 3
4. 5
5. 7
6. 14
7. 30
8. 60
9. 90
10. 120
11. 150
12. 180
13. 365
14. 400
15. 545
16. 731
17. 1096
18. 1827
19. 2192
20. 2557
21. 2922
22. 3288
23. 3653
default choice: [1]:
7
Do you want to specify any additional log files to monitor?
1. yes
2. no
default choice: [1]:
2
Do you want the CloudWatch agent to also retrieve X-ray traces?
1. yes
2. no
default choice: [1]:
2
Existing config JSON identified and copied to:  /opt/aws/amazon-cloudwatch-agent/etc/backup-configs
Saved config file to /opt/aws/amazon-cloudwatch-agent/bin/config.json successfully.
Current config as follows:
{
        "agent": {
                "metrics_collection_interval": 60,
                "run_as_user": "root"
        },
        "logs": {
                "logs_collected": {
                        "files": {
                                "collect_list": [
                                        {
                                                "file_path": "/var/log/messages",
                                                "log_group_class": "STANDARD",
                                                "log_group_name": "tinnakorn-cwagent-ec2/var_log_messages",
                                                "log_stream_name": "{instance_id}",
                                                "retention_in_days": 30
                                        }
                                ]
                        }
                }
        },
        "metrics": {
                "aggregation_dimensions": [
                        [
                                "InstanceId"
                        ]
                ],
                "append_dimensions": {
                        "AutoScalingGroupName": "${aws:AutoScalingGroupName}",
                        "ImageId": "${aws:ImageId}",
                        "InstanceId": "${aws:InstanceId}",
                        "InstanceType": "${aws:InstanceType}"
                },
                "metrics_collected": {
                        "collectd": {
                                "metrics_aggregation_interval": 60
                        },
                        "disk": {
                                "measurement": [
                                        "used_percent"
                                ],
                                "metrics_collection_interval": 60,
                                "resources": [
                                        "*"
                                ]
                        },
                        "mem": {
                                "measurement": [
                                        "mem_used_percent"
                                ],
                                "metrics_collection_interval": 60
                        },
                        "statsd": {
                                "metrics_aggregation_interval": 60,
                                "metrics_collection_interval": 10,
                                "service_address": ":8125"
                        }
                }
        }
}
Please check the above content of the config.
The config file is also located at /opt/aws/amazon-cloudwatch-agent/bin/config.json.
Edit it manually if needed.
Do you want to store the config in the SSM parameter store?
1. yes
2. no
default choice: [1]:
2
Program exits now.
[root@ip-xx-xx-xx-xx ~]#

sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -s -c file:/opt/aws/amazon-cloudwatch-agent/bin/config.json

Configuration validation second phase failed
======== Error Log ========
2024-08-14T03:58:50Z E! [telegraf] Error running agent: Error loading config file /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.toml: error parsing socket_listener, open /usr/share/collectd/types.db: no such file or directory

sudo systemctl start amazon-cloudwatch-agent.service

sudo systemctl status amazon-cloudwatch-agent.service

sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -m ec2 -a status

sudo systemctl stop amazon-cloudwatch-agent.service

sudo journalctl -u amazon-cloudwatch-agent.service

tail -f /var/log/messages